Solutions to Exercises of Chapters 3, 4 and 5

CHAPTER 3

3.1 The σ-algebra σ(X) consists of all subsets of X(Ω). A set with n elements has precisely 2^{n} subsets. Hence σ(X) consists of 2^{n} sets. Each of these sets can be obtained as follows: Let x_{1}, …, x_{n} be the distinct values of X. Enumerate the binary strings of 0’s and 1’s of length n and to each of them assign a set from σ(X). The way to do this should be clear from the following example with n = 3:

\begin{matrix}000 : & \{X ≠ x_{1}, X ≠ x_{2}, X ≠ x_{3}\} = ∅ \\ 001 : & \{X ≠ x_{1}, X ≠ x_{2}, X = x_{3}\} = \{X = x_{3}\} \\ 010 : & \{X ≠ x_{1}, X = x_{2}, X ≠ x_{3}\} = \{X = x_{2}\} \\ 011 : & \{X ≠ x_{1}, X = x_{2}, X = x_{3}\} = \{X = x_{2} or x_{3}\} \\ 100 : & \{X = x_{1}, X ≠ x_{2}, X ≠ x_{3}\} = \{X = x_{1}\} \\ 101 : & \{X = x_{1}, X ≠ x_{2}, X = x_{3}\} = \{X = x_{1} or x_{3}\} \\ 110 : & \{X = x_{1}, X = x_{2}, X ≠ x_{3}\} = \{X = x_{1} or x_{2}\} \\ 111 : & \{X = x_{1}, X = x_{2}, X = x_{3}\} = Ω \end{matrix}

3.2 (i) If x_{1} < x_{2} then {X ≤ x_{1}} ⊂ {X ≤ x_{2}}. (ii) lim_{x→−∞} F (x) = lim_{n→−∞} F (n), where n ranges over the integers. But F (n) = P(X ≤ n) decreases as n decreases, and so

lim_{n→−∞} F (n) = P ( ⋂_{n∈ℤ}{X ≤ n}) = P(∅) = 0.

(iii) Similarly,

lim_{n→+∞} F (n) = P ( ⋃_{n∈ℤ}{X ≤ n}) = P(Ω) = 1.

(iv)

⋂_{n∈ℕ}(−∞, x + 1/n] = (−∞, x].

3.3 For a ≤ b: (i)

(a, b] = (−∞, b] \ (−∞, a]

and (−∞, a] ⊂ (−∞, b]. (ii) Observe that

(a, b) = ⋃_{n∈ℕ}(a, b − 1/n]

and use (i) to get

P(X ∈ (a, b)) = lim_{n→∞}(F (b − 1/n) − F (a)) = F (b−) − F (a).

(iii) Observe that

[a, b] = ⋂_{n∈ℕ}(a − 1/n, b]

and use (i) to get

P(X ∈ [a, b]) = lim_{n→∞}(F (b) − F (a − 1/n)) = F (b) − F (a−).

(iv) Apply (iii) with a = b.

3.4 The way that F^{−1} is defined in the lecture notes is:

F^{−1}(t) := inf A_{t}, where A_{t} := {x ∈ ℝ : F (x) ≥ t}.

By the definition of the infimum of a set of real numbers (the set A_{t}, in this case), F^{−1}(t) is the largest of all numbers c which are lower bounds to the set (the existence of which is guaranteed by the completeness property of the set of real numbers–a consequence of its construction):

F^{−1}(t) ≥ c ⟺ x ≥ c for all x ∈ A_{t}.

Equivalently,

F^{−1}(t) ≥ c ⟺ for all x < c, x ∉ A_{t} ⟺ for all x < c, F (x) < t.

Instead of saying “for all x < c” we can say “for all x = c − ε with ε > 0”, and so

F^{−1}(t) ≥ c ⟺ F (c − ε) < t for all ε > 0 ⟺ F (c−) < t.

Therefore,

F^{−1}(t) < c ⟺ t ≤ F (c−),

This holds for all t and c and so, by setting t = U, we obtain

P(F^{−1}(U) < c) = P(U ≤ F (c−)) = F (c−),

the latter due to the assumption that U is uniformly distributed on the interval (0, 1). Hence

P(F^{−1}(U) ≤ x) = lim_{n→∞} P(F^{−1}(U) < x+1/n) = lim_{n→∞} F ((x+1/n)−) = lim_{n→∞} F (x+1/n) = F (x).

3.5 The function V : {0, 1}^{ℕ} → ℝ is defined by

V (ω) = ∑_{k=1}^{∞} 2ω_{k}3^{−k}, ω_{1}, ω_{2}, … ∈ {0, 1},

and P is a measure on the cylinder-σ-algebra ℱ of {0, 1}^{ℕ} such that

P(ω_{1} = ε_{1}, …, ω_{n} = ε_{n}) = 2^{−n},

for all n ∈ ℕ, and all ε_{1}, …, ε_{n} ∈ {0, 1}. We consider the distribution function

F (x) = P(V ≤ x)

of V. Define V′(ω) by

V (ω) = \frac{2ω_{1}}{3} + \frac{1}{3}V′(ω).

Observe that the P-law of V′ is the same as the P-law of V (by considering finite-dimensional cylinder sets). Suppose x < 1/3. Then

F (x) = P(V ≤ x) = P(ω_{1} = 0, \frac{1}{3}V′ ≤ x) = \frac{1}{2}F (3x).

Suppose x > 2/3. Then

F (x) = P(V ≤ x) = P(ω_{1} = 1, \frac{2}{3} + \frac{1}{3}V′ ≤ x) = \frac{1}{2}F (3(x − \frac{2}{3})).

Observe that it is impossible for V (ω) to be strictly between 1/3 and 2/3. Indeed, if ω_{1} = 0, then the maximum value of V (ω) is 1/3 (and this happens when V′(ω) = 1; if ω_{1} = 1, the least value of V (ω) is 2/3. So

{1/3 < V < 2/3} = ∅.

Since

P(V = 1/3) = P(ω_{1} = 0, V′ = 1) = P(ω_{1} = 0, ω_{n} = 1 for all n ≥ 1) = 0,
P(V = 2/3) = P(ω_{1} = 1, V′ = 0) = P(ω_{1} = 1, ω_{n} = 0 for all n ≥ 1) = 0,

we have

P(1/3 ≤ V ≤ 2/3) = 0

and so, for some constant c,

F (x) = c, if 1/3 ≤ x ≤ 2/3.

Since V and 1 − V have the same law we have

c = 1/2.

In other words, if we define the operator Q on [0, 1]^{[0,1]} by

Qf(x) = \begin{cases}\frac{1}{2}f(3x), & if 0 ≤ x < 1/3 \\ 1/2, & if 1/3 ≤ x ≤ 2/3 \\ \frac{1}{2}f(3(x − 2/3)), & if 2/3 < x ≤ 1, \end{cases}

we have just shown that F satisfies

F = QF.

Now restrict Q onto C[0, 1], the space of continuous functions on [0, 1], equipped with the usual norm

||f|| = max_{0≤x≤1} |f(x)|.

Observe that

||Qf|| = max_{0≤x≤1} |f(x)| = max_{0≤x<1/3} |f(x)| ∨ max_{1/3≤x≤2/3} |f(x)| ∨ max_{2/3<x≤1} |f(x)|

= max_{0≤x<1/3} |\frac{1}{2}f(3x)| ∨ (1/2) ∨ max_{2/3<x≤1} |\frac{1}{2}f(3(x − 2/3))| = \frac{1}{2}||f||.

So Q is a contraction, and so, by completeness of C[0, 1] (this is the analogous of the completeness as the completeness mentioned in Exercise 3.4), starting from any F_{0} ∈ C[0, 1], the sequence defined recursively through

F_{n+1} = QF_{n}

converges uniformly to an F^{∗} ∈ C[0, 1] which satisfies

QF^{∗} = F^{∗}.

Note that Q is an increasing operator, i.e. if f is increasing function, then so is Qf. Therefore, the limit is also increasing. So F^{∗} is an increasing continuous function. By observing that Q is a contraction also on the set D[0, 1] of functions on [0, 1] which have discontinuities of first kind, and using the fact that F ∈ D[0, 1], we have

F = F^{∗}

and so F is continuous. To show that F is not absolutely continuous, we show that

P(V ∈ C) = 1

for some set C ⊂ [0, 1] with Lebesgue measure 0. Let H be a nonempty open subset of [0, 1] such that

P(V ∈ H) = 0.

(We already saw that H = (1/3, 2/3) is such a set.) Notice that

\frac{1}{3}H = \{x/3 : x ∈ H\} ⊂ (0, 1/3),

\frac{2}{3} + \frac{1}{3}H = \{(2/3) + x/3 : x ∈ H\} ⊂ (2/3, 1).

Hence

P(V ∈ \frac{1}{3}H) = P(ω_{1} = 0, \frac{1}{3}V′ ∈ \frac{1}{3}H) = \frac{1}{2} P(V ∈ H)

P(V ∈ \frac{2}{3} + \frac{1}{3}H) = P(ω_{1} = 1, \frac{2}{3} + \frac{1}{3}V′ ∈ \frac{2}{3} + \frac{1}{3}H) = \frac{1}{2} P(V ∈ H)

So, starting with a set H_{0} = H for which P(V ∈ H) = 0, we can create a family of sets H_{n} for which P(V ∈ H_{n}) = 0, recursively by

H_{n+1} = \frac{1}{3}H_{n} ∪ (\frac{2}{3} + \frac{1}{3}H_{n}), n = 1, 2, …

Since the sets \frac{1}{3}H_{n} and \frac{2}{3} + \frac{1}{3}H_{n} are disjoint nonempty open sets, we have that their lengths add up. But both sets have length equal to a third of the length of H_{n}. So

|H_{n+1}| = \frac{2}{3}|H_{n}|,

i.e.

|H_{n}| = (2/3)^{n}|H_{0}|.

Let

D = ∪_{n=0}^{∞}H_{n},

where H_{0} = (1/3, 2/3). We have

P(V ∈ D) ≤ ∑_{n=0}^{∞} P(V ∈ H_{n}) = ∑_{n=0}^{∞} 0 = 0.

We have |H_{0}| = 1/3, |H_{n}| = 2^{n}/3^{n+1}. Observe that the H_{n} are here disjoint. So

|D| = ∑_{n=0}^{∞} |H_{n}| = ∑_{n=0}^{∞} 2^{n}/3^{n+1} = 1.

Finally, set

C = [0, 1] − D.

We have |C| = 1 − |D| = 0, and P(V ∈ C) = 1 − P(V ∈ D) = 1 − 0 = 1.

3.6 Y = 1 if and only if X divides 3, and the only possibility is X = 3. So P(Y = 1) = 1/6, P(Y = 2) = 5/6.

3.7

P(φ(X) ≤ t) = P(X ≥ ψ(t)) = ∫_{ψ(t)}^{∞} f(x)dx.

Taking derivative with respect to t, we find that the density of φ(X) equals

\frac{d}{dt} ∫_{ψ(t)}^{∞} f(x)dx = −ψ′(t)f(ψ(t)).

But ψ is strictly decreasing, so ψ′(t) < 0, and so −ψ′(t) = |ψ′(t)|.

3.8 Let a > 0. The inverse of the function φ(u) := u^{1/a} is ψ(y) = y^{a}. We have ψ′(y) = ay^{a−1}. So

f_{Y} (y) = f_{U}(ψ(y))ψ′(y) = ay^{a−1}.

Clearly, 0 < y < 1 is the range over which f is ≠ 0, because φ maps the interval (0, 1) to itself. If a = −b < 0, then φ(u) := u^{−1/b} maps (0, 1) into (1, ∞). So Y has nonzero density on (1, ∞) and, by the exercise above, the density is given by

f_{Y} (y) = f_{U}(ψ(y))|ψ′(y)| = by^{−b−1}.

You should sketch the functions φ and ψ in both cases, and see why, qualitatively, the results are sound, i.e. that the mass gets transferred to the correct places. E.g., the function Y = U^{30} has most of its density around 0, while Y = U^{1/30} has most of its density around 1.

3.9 Let us write the solution in intuitive terms (using calculus), following, of course, a method which is totally equivalent to the theory. Let u be mapped to y via y = e^{u}. Then the mass assigned on an interval of tiny length |du| sitting around the point u is transferred to an interval of tiny length |dy| sitting around the point y:

f_{Y} (y)|dy| = f_{U}(u)|du|

and, since \frac{dy}{du} = e^{u}, we have

f_{Y} (y)e^{u} = f_{U}(u),

or

f_{Y} (y) = e^{−u} = e^{− log y} = \frac{1}{y}.

The range of interest is the image of the interval (0, 1) under the map y = e^{u}, i.e. y ranges over (e^{0}, e^{1}) = (1, e). Let us now consider the case y = e^{au}, where a is a real number. The function is one-to-one for all values of a, so we need not worry about multiple pre-images of a point. Again,

f_{Y} (y)|dy| = f_{U}(u)|du|

Here, \frac{dy}{du} = ae^{au}, and so

f_{Y} (y)|ae^{au}| = f_{U}(u),

i.e.

f_{Y} (y) = \frac{1}{|a|e^{au}} .

Let us not forget that we need to express this is a function of y, through y = e^{au}, i.e.

f_{Y} (y) = \frac{1}{|a|y} .

The range of y is the image of (0, 1) under the map y = e^{au}. If a > 0, then y ranges over (1, e^{a}). If a < 0, then y ranges over (e^{−|a|}, 1). If a = 0 we have a degenerate situation, because Y = 1 with probability 1 (no density).

3.10 (Note there are some typos in the statement of the exercise.) No sweat here: the method is as above:

f_{Y} (y)|dy| = f_{X}(x)|dx|

We have y = e^{x/a}, where a > 0, so \frac{dy}{dx} = \frac{1}{a}e^{x/a}:

f_{Y} (y)\frac{1}{a}e^{x/a} = f_{X}(x),

f_{Y} (y) = e^{−x}ae^{−x/a} = ay^{−a−1}.

Here y ranges between 1 and ∞ because y = e^{x/a} maps {x > 0} onto {1 < y < ∞}.

3.11 If the function y = φ(x) is non-monotonic, then each y may have multiple pre-images. So an interval of tiny length |dy| located around the point y may be the image of many tiny intervals located at the points x_{1}, x_{2}, …, where the latter are all pre-images of y under the map y = φ(x). Hence the “mass” f_{Y} (y)|dy| is the sum of the masses f_{X}(x_{i})|dx|, i = 1, 2, …:

f_{Y} (y)|dy| = ∑_{i} f_{X}(x_{i})|dx|.

Let us consider the case Y = X^{2}, where X is uniform on [−1, 1]. Then \frac{dy}{dx} = 2x. There are 2 solutions of the equation y = x^{2}, namely, x_{1} = +\sqrt{y}, x_{2} = −\sqrt{y}. So

f_{Y} (y) = f_{X}(\sqrt{y}) |\frac{dx}{dy}|_{x=\sqrt{y}} + f_{X}(−\sqrt{y}) |\frac{dx}{dy}|_{x=−\sqrt{y}} .

But f_{X}(x) = 1/2, for −1 ≤ x ≤ 1 and |\frac{dx}{dy} |_{x=\sqrt{y}} = |\frac{dx}{dy} |_{x=−\sqrt{y}} = 1/(2\sqrt{y}), so

f_{Y} (y) = 1/(2\sqrt{y}), 0 ≤ y ≤ 1.

Alternatively, we can argue directly as follows:

P(X^{2} ≤ y) = P(−\sqrt{y} ≤ X ≤ \sqrt{y}).

So

f_{Y} (y) = \frac{d}{dy} P(X^{2} ≤ y) = \frac{d}{dy} ∫_{−\sqrt{y}}^{√\overline{y}} \frac{1}{2}dy′ = \frac{1}{2\sqrt{y}} .

3.11 In the second case,

Y = X^{2}, where X is uniform on [-1,2]

every y ∈ (0, 1) has two pre-images, x = ±\sqrt{y}, but every y ∈ (1, 4) has one pre-image, x = +\sqrt{y}. So

f_{Y} (y) = \begin{cases}f_{X}(\sqrt{y}) |\frac{dx}{dy} |_{x=\sqrt{y}} + f_{X}(−\sqrt{y}) |\frac{dx}{dy} |_{x=−\sqrt{y}} , & 0 < y < 1, \\ f_{X}(\sqrt{y}) |\frac{dx}{dy} |_{x=\sqrt{y}} , & 1 < y < 4. \end{cases}

Here f_{X}(x) = 1/3, −1 ≤ x ≤ 2 and |\frac{dx}{dy} | = 1/(2\sqrt{y}) in all cases, so

f_{Y} (y) = \begin{cases}1/(3\sqrt{y}), & 0 < y < 1, \\ 1/(6\sqrt{y}), & 1 < y < 4. \end{cases}

The density has a jump at y = 1, and we don’t bother to define its value there: it is irrelevant because the density is useful only as an object to be integrated. In other words, not defining the value of a density at finitely many points won’t matter at all.

3.12 Same story here:

f_{Y} (y)|dy| = (f_{X}(x_{1}) + f_{X}(x_{2}))|dx|,

where x_{1}, x_{2} are the two pre-images of y = cosh(x). Here y ranges over [1, ∞), and |\frac{dy}{dx}| = | sinh(x)| = \sqrt{y^{2} − 1}. The latter follows from the identity cosh^{2}(x) − sinh^{2}(x) = 1. Notice that |x_{1}| = |x_{2}| = cosh^{−1}(y) and the density of X is symmetric around zero. Hence

f_{Y} (y) = \frac{2c}{1 + cosh^{−1}(y)^{2}} \frac{1}{\sqrt{y^{2} − 1}} = \frac{c}{\sqrt{y^{2} − 1}y^{2} + y^{3} − y}

3.13 We have

P(X = a^{n}) = 2^{−n}, n ∈ ℕ.

Therefore,

E X = ∑_{n=1}^{∞}(a/2)^{n}

which is finite iff a/2 < 1. The sum equals a/(2 − a).

3.14 We have

P(X = n^{k}) = cn^{−2}, n ∈ ℕ.

Therefore,

E X = ∑_{n=1}^{∞} cn^{−(2−k)},

and the sum is finite iff 2 − k > 1. (Use, e.g., the ratio test.)

3.15

E(X; A) = ∑_{\begin{matrix}n∈ℕ \\ n odd \end{matrix}}(a/2)^{n} = ∑_{m=0}^{∞}(a/2)^{2m+1} = \frac{2a}{a^{2} − 4}.

Setting a = 1 in the above we obtain

P(A) = ∑_{m=0}^{∞}(1/2)^{2m+1} = \frac{2}{3},

so E(X|A) = E(X; A)/ P(A) = 3a/(a^{2} − 4). The reason that E(X|A^{c}) = a E(X|A) is obvious is that X = a^{ξ}, where P(ξ = n) = 2^{−n}, n ∈ ℕ, i.e. ξ is a geometric–and hence memoryless–random variable.

3.16 The solution depends on the way that theory was presented.

3.17

E N = ∑_{n=1}^{∞} ne^{−λ} \frac{λ^{n}}{n!} = e^{−λ}λ ∑_{n=1}^{∞} \frac{λ^{n−1}}{(n − 1)!} = e^{−λ}λe^{λ} = λ,

where we used the Taylor expansion of the exponential function.

E(N^{2} − N) = ∑_{n=2}^{∞}(n^{2} − n)e^{−λ} \frac{λ^{n}}{n!} = e^{−λ}λ^{2} ∑_{n=2}^{∞} \frac{λ^{n−2}}{(n − 2)!} = e^{−λ}λ^{2}e^{λ} = λ^{2}.

Hence E(N^{2}) = E(N^{2} − N) + E N = λ^{2} + λ.

CHAPTER 4

4.1 We use Chernoff’s inequality as follows:

P(X > na) ≤ \frac{E e^{θX}}{e^{θna}} ,

where θ is a positive constant. Now,

E e^{θX} = ∑_{k=0}^{n} e^{θk}\binom{n}{k}2^{−k} = ∑_{k=0}^{n} \binom{n}{k}(e^{θ}2^{−1})^{k} = (1 + e^{θ}2^{−1})^{n}.

Hence,

P(X > na) ≤ (\frac{1 + e^{θ}2^{−1}}{e^{aθ}} )^{n} ,

and the bound holds for any θ > 0. We can get a value for the bound by choosing some specific θ, e.g. θ = log 4, which gives (3 × 4^{−a})^{n}, or we can be smart and find the best bound by choosing the value of θ for which the bound is least. To find this θ, take the logarithm of the expression inside the parenthesis, differentiate with respect to θ, and set the derivative equal to zero:

0 = \frac{d}{dθ} (log(1 + e^{θ}2^{−1}) − aθ) = \frac{e^{θ}2^{−1}}{1 + e^{θ}2^{−1}} − a,

whence, e^{θ} = \frac{2a}{1 − a}.

For this value, we have

P(X > na) ≤ (\frac{1 + \frac{a}{1−a}}{(\frac{2a}{1−a})^{a}} )^{n} = 2^{−an}a^{−an}(1 − a)^{−(1−a)n}

4.2 This is an exercise about a nonnegative random variable Z for which we know that E Z = 0. If Z is simple, then it is immediate that P(Z > 0) = 0. For general Z, let Z_{n} be a sequence of simple random variables such that Z_{n} ↑ Z. Then {Z > 0} = ∪_{n}{Z_{n} > 0} and since the latter events have P equal to zero, it follows that P(Z > 0) = 0. Now apply this to the random variable Z = (X − E X)^{2}. We learnt that P(Z = 0) = 1, i.e. P(X = E X) = 1.

4.3 Let Z = |X − E X|^{2}. Markov’s inequality says

P(Z > t^{2}) ≤ \frac{E Z}{t^{2}} ,

for all t > 0. But

P(Z > t^{2}) = P(|X − E X| > t).

4.4 Just use the binomial theorem:

(a(X − E X) + b(Y − E Y))^{2} = a^{2}(X − E X)^{2} + 2ab(X − E X)(Y − E Y) + b^{2}(Y − E Y)^{2} 4.5 Again, use the binomial theorem:

(λX + Y)^{2} = λ^{2}X^{2} + 2λXY + Y^{2}.

Hence

Q(λ) := λ^{2} E X^{2} + 2λ E XY + E Y^{2} ≥ 0

for all values of λ. The polynomial Q(λ) is minimised at the λ which solves

0 = Q′(λ) = 2λ E X^{2} + 2 E XY,

i.e. for λ = − E XY/ E X^{2}. Hence

0 ≤ Q(− E XY/ E X^{2}) = \frac{E X^{2} E Y^{2} − (E XY)^{2}}{E X^{2}} ,

which gives the inequality we need.

4.6 Apply the previous inequality to X − E X, Y − E Y in lieu of X, Y, respectively.

4.7 Just use the property e^{x+y} = e^{x}e^{y} of the exponential function.

e^{θ(a+bX)} = e^{θa}e^{θbX}
E e^{θ(a+bX)} = e^{θa} E e^{θbX} = e^{θa}M_{X}(θb).

4.8 If λ > θ,

M(θ) = ∫_{−∞}^{∞} e^{θx}λe^{−λx}dx = λ ∫_{−∞}^{∞} e^{−x(λ−θ)}dx = \frac{λ}{λ − θ} .

If λ ≤ θ the integral diverges. Now,

M′(θ) = \frac{λ}{(λ − θ)^{2}} , M′′(θ) = \frac{2λ}{(λ − θ)^{3}} ,

So,

E X = M′(0) = \frac{1}{λ}, E X^{2} = M′′(0) = \frac{2}{λ^{2}} , var X = E X^{2} − (E X)^{2} = \frac{1}{λ^{2}} .

4.9 A necessary condition for the integral defining M(θ), for θ > 0, to converge, is that the right tail of the density decays exponentially. This is not so for the Cauchy distribution and so M(θ) = ∞ for θ > 0. By symmetry, M(θ) = M(−θ) for θ < 0, and so M(θ) = ∞, for all values of θ ≠ 0.

4.10 Let Y_{1}, Y_{2} be real random variables. The modulus inequality is equivalent to

\sqrt{E Y_{1}^{2} + E Y_{2}^{2}} ≤ E \sqrt{Y_{1}^{2} + Y_{2}^{2}}.

But, by Jensen’s inequality, the square of the right hand side is larger than or equal to E(Y_{1}^{2} + Y_{2}^{2}), which proves the inequality.

4.11 The exponential function satisfies e^{z+w} = e^{z}e^{w} for any two complex numbers z, w.

e^{it(a+bX)} = e^{ita}e^{itbX}

E e^{it(a+bX)} = e^{ita} E e^{itbX} = e^{ita}φ_{X}(tb).

4.12

∫_{0}^{1} e^{itu}du = \frac{e^{it} − 1}{it}.

If X is uniform on (a, b) then X = (b − a)U + a, where U is uniform on (0, 1). 4.13 We need to show that

∫_{0}^{∞} e^{iθx}λe^{−λx}dx = \frac{λ}{λ − iθ} .

Note that the function f(z) = e^{z} is analytic for all z ∈ ℂ, and has itself as primitive. So

∫_{γ} e^{z}dz = e^{b} − e^{a},

for any simple curve γ with endpoints a, b ∈ ℂ. Let γ be the curve z = −cx, 0 ≤ x ≤ x_{0}, where c is a fixed complex number. Then

∫_{γ} e^{z}dz = ∫_{0}^{x_{0}} e^{−cx}(−c)dx.

Hence

∫_{0}^{x_{0}} e^{−cx}dx = \frac{1 − e^{−cx_{0}}}{c} .

If the real part of c is positive, lim_{x_{0}→∞} e^{−cx_{0}} = 0, and so

∫_{0}^{∞} e^{−cx}dx = \frac{1}{c},

which gives what we need if we set c = λ − iθ. 4.14 For x ≤ y, both ranging in {1, …, 6},

P(X ≥ x, Y ≤ y) = P(x ≤ N_{1} ≤ y, x ≤ N_{2} ≤ y) = (\frac{y − x}{6})^{2}.

The marginals are as follows:

P(X ≥ x) = (\frac{6 − x}{6})^{2}

P(Y ≤ y) = (\frac{y}{6})^{2}

We can work out P(X = x, Y = y) by using the additivity of P, or, simply, by

P(X = x, Y = y) = 2 P(N_{1} = x, N_{2} = y) = 1/6, x ≤ y.

To find the conditional probabilities, just use division.

4.15 Just do an integral using Fubini:

a^{−1} = ∫\begin{matrix}0≤x≤1 \\ 0≤y≤1 \end{matrix}(x + y^{2})d(x, y) = ∫_{0≤x≤1} dx ∫_{0≤y≤1} dy(x + y^{2}) = \frac{1}{2} + \frac{1}{3}.

F_{X,Y} (x, y) = a ∫\begin{matrix}0≤x′≤x \\ 0≤y′≤y \end{matrix}(x′ + y^{′2})d(x′, y′) = a\frac{x^{2}}{2} y + ax\frac{y^{3}}{3} .

P(X > Y) = ∫\begin{matrix}0≤x≤1 \\ 0≤y≤1 \\ x>y \end{matrix} a(x + y^{2}) = a ∫_{0≤x≤1} dx ∫_{0≤y≤1} dy1(x > y)(x + y^{2})

= a ∫_{0≤x≤1} x^{2}dx + a ∫_{0≤y≤1}(1 − y)y^{2}dy = a(\frac{1}{3} + \frac{1}{3} − \frac{1}{4}).

P(X^{2} > Y) = ∫\begin{matrix}0≤x≤1 \\ 0≤y≤1 \\ x^{2}>y \end{matrix} a(x + y^{2}) = a ∫_{0≤x≤1} dx ∫_{0≤y≤1} dy1(x^{2} > y)(x + y^{2})

= a ∫_{0≤x≤1} x^{3}dx + a ∫_{0≤x≤1} \frac{(x^{2})^{3}}{3} dx = a(\frac{1}{4} + \frac{1}{3·7}).

4.16 We have that X, W are independent exponentials with E X = 2, E W = 1, and Y = \frac{1}{2}X − W. As noted, Y ≥ \frac{1}{2}X, with probability 1. We compute the distribution function in the form

G(x, y) := P(X > x, Y > y),

because it’s easier to do the integrals. The variable y ranges from −∞ to +∞ because Y can take negative values.

G(x, y) = P(X > x, X > W + 2y)
= ∫_{0}^{∞} dw e^{−w} P(X > x, X > 2w + 2y) = ∫_{0}^{∞} dw e^{−w} P(X > x ∨ (2w + 2y))
= ∫_{0}^{∞} dw e^{−w} e^{−[(x/2)∨(w+y)]}

This is an easy integral in the variable w, as long as we split it into two integrals: one over w < (x/2) − y and one over w > (x/2) − y. We here assume (I) that x/2 > y (otherwise the first integral is vacuus.)

G(x, y) = ∫_{0}^{(x/2)−y} dw e^{−w} e^{−x/2} + ∫_{(x/2)−y}^{∞} dw e^{−w} e^{−w−y}

= e^{−x/2} ∫_{0}^{(x/2)−y} dw e^{−w} + e^{−y} ∫_{(x/2)−y}^{∞} dw e^{−2w}

= e^{−x/2}(1 − e^{− \frac{x}{2} +y}) + \frac{1}{2}e^{−y}e^{−x+2y} = e^{−x/2} − \frac{1}{2}e^{−x+y}.

If (II) x/2 < y then (x/2) ∨ (w + y) = w + y for all w > 0, and so

G(x, y) = ∫_{0}^{∞} dw e^{−w} e^{−w−y} = \frac{1}{2}e^{−y}.

So the answer is:

P(X > x, Y > y) = \begin{cases}e^{−x/2} − \frac{1}{2}e^{−x+y}, & if x ≥ 2y \\ \frac{1}{2}e^{−y}, & if x ≤ 2y. \end{cases}

We can also find F_{X,Y} (x, y) using additivity:

1 − G(x, y) = P(X ≤ x or Y ≤ y) = 1 − P(X > x) + 1 − P(Y > y) − F_{X,Y} (x, y). 4.17

E e^{ηX′+θY′} = E e^{η(X+Y)+θ(X−Y)} = E e^{(η+θ)X+(η−θ)Y}
= E e^{(η+θ)X} E e^{(η−θ)Y}
= e^{(η+θ)^{2}}e^{(η−θ)^{2}}
= e^{2η^{2}+2θ^{2}}.

Since

e^{2η^{2}+2θ^{2}} = e^{2η^{2}}e^{2θ^{2}},

X′, Y′ are independent, both normal with zero mean and variance 2. 4.18

∑_{n=0}^{∞} e^{−λ} \frac{λ^{n}}{n!} (e^{θ}n) = e^{−λ} ∑_{n=0}^{∞} \frac{(λe^{θ})^{n}}{n!} = e^{−λ}e^{λe^{θ}} = e^{λ(e^{θ}−1)}.

See Exercise 5.2 below.

CHAPTER 5

5.1 This is the same as Exercise 4.1, except that there p was equal to 1/2. The method is precisely the same. 5.2 We use generating functions:

∏_{j=1}^{n} M_{X_{j}}(θ) = ∏_{j=1}^{n} exp (λ_{j}(e^{θ} − 1)) = exp ( ∑_{j=1}^{n} λ_{j}(e^{θ} − 1)),

and the latter is the generating function of a Poisson random variable with rate ∑_{j} λ_{j}. 5.3 A brute-force proof is as follows:

P(X_{1} = n_{1}, …, X_{d} = n_{d} | X_{1} + ⋯ + X_{d} = n) = \frac{P(X_{1} = n_{1}, …, X_{d} = n_{d})}{P(X_{1} + ⋯ + X_{d} = n)}

= \frac{∏_{k=1}^{d} \frac{λ_{k}^{n_{k}}}{n_{k}!} e^{−λ_{k}}}{\frac{λ^{n}}{n!} e^{−λ}} = \frac{n!}{∏_{k=1}^{d} n_{k}!} ∏_{k=1}^{d}(λ_{k}/λ)^{n_{k}} = \binom{n}{n_{1}, …, n_{d}} ∏_{k=1}^{d}(λ_{k}/λ)^{n_{k}}.

(A proof can also be devised by using Exercise 5.8 below.) 5.4 Since

P(X > k|X > k − 1) = q

we have

P(X > k) = q P(X > k − 1) = q^{2} P(X > k − 2) = ⋯ = q^{k},

and so X is geometric. The values P(X = 1) is called parameter. So the parameter is P(X = 1) = P(X > 0) − P(X > 1) = q^{0} − q^{1} = 1 − q. 5.5

M(θ) = E e^{θX} = ∑_{n=1}^{∞} e^{θn}(1 − p)^{n−1}p = pe^{θ} ∑_{n=1}^{∞}((1 − p)e^{θ})^{n−1} = \frac{pe^{θ}}{1 − (1 − p)e^{θ}} .

E X = M′(0) = 1/p, E X^{2} = M′′(0) = (2 − p)/p^{2}, var X = (1 − p)/p^{2}. 5.6 We have

P(X > Y + n, X > Y) = ∑_{k} P(X > k + n, X > k, Y = k)

= ∑_{k} P(X > k + n|X > k) P(X > k) P(Y = k)

= P(X > n) ∑_{k} P(X > k) P(Y = k)

= P(X > n) P(X > Y).

Dividing by P(X > Y) we obtain the result. The result can be interpreted as follows: If X represents the duration of my sleep (which is geometrically distributed) then: given that I have not waken up by the unknown time Y that an explosion will occur in Australia, my remaining sleeping time X − Y will be distributed as X, i.e. as if the explosion occur ed when I went to bed.

5.7 From the formula of density transformation, it is obvious that cX + d has constant density. Since the function y = cx+d maps the interval [a, b] onto the interval with endpoints ca + d and cb + d, the result follows.

5.8 We can get the answer easily if we think combinatorially. We want to compute

P(A) ≡ P(S_{n}^{1} = m_{1}, …, S_{n}^{d} = m_{n}),

for all non-negative integers m_{1}, …, m_{n} adding up to n. The probability that the first m_{1} of the U_{j}’s fall in I_{1} and the next m_{2} of them in I_{2}, and so on, equals p_{1}^{m_{1}} ⋯ p_{d}^{m_{d}}. The event whose probability we just computed is one of the many events comprising A. Each of these events has exactly the same probability and there are \binom{n}{m_{1},...,m_{d}} such events. Therefore,

P(A) = \binom{n}{m_{1}, …, m_{d}}p_{1}^{m_{1}} ⋯ p_{d}^{m_{d}}.

For the analytically minded, we can verify (and prove) that the result is correct by checking that the generating functions of both sides are equal. First, for the multinomial, we have

∑ \binom{n}{m_{1}, …, m_{d}}p_{1}^{m_{1}} ⋯ p_{d}^{m_{d}}θ_{1}^{m_{1}} ⋯ θ_{d}^{m_{d}} = (p_{1}θ_{1} + ⋯ + p_{d}θ_{d})^{n},

where the sum extends over all non-negative integers m_{1}, …, m_{n} adding up to n, and where we have used the multinomial theorem. Second, for the probability we are seeking, we have

∑ P(S_{n}^{1} = m_{1}, …, S_{n}^{d} = m_{n})θ_{1}^{m_{1}} ⋯ θ_{d}^{m_{d}} = E [θ_{1}^{S_{n}^{1}} ⋯ θ_{d}^{S_{n}^{d}}],

where the sum extends over the same region as before. The latter further equals

E ∏_{r=1}^{d} ∏_{j=1}^{n} θ_{r}^{1(U_{j}∈I_{r})} = E ∏_{j=1}^{n} ∏_{r=1}^{d} θ_{r}^{1(U_{j}∈I_{r})} = ∏_{j=1}^{n} E ∏_{r=1}^{d} θ_{r}^{1(U_{j}∈I_{r})}

The function inside the expectation is a function of U_{j} only and is a simple function: it takes value θ_{r} with probability p_{r}; therefore it has expectation ∑_{r=1}^{d} p_{r}θ_{r}. Hence the two generating functions agree.

5.9 First note that, with probability 1, all random variables are distinct. There are d! ways to order U_{1}, …, U_{d}. Since the probabilities of each order are equal and since the probabilities add up to 1, it follows that the probability of a specific order is 1/d!. So P(U_{1} < ⋯ < U_{d}) = 1/d!.

5.10 If X, Y, Z are the lengths of the sticks then X + Y + Z = 1, and the sticks form a triangle^{1} if (from Euclidean Geometry) each of the sticks has length smaller than the sum of the lengths of the other two:

X ≤ Y + Z
Y ≤ Z + X
Z ≤ X + Y.

The model for stick breaking is, undoubtedly, as follows: Let the stick be the interval [0, 1] = {x ∈ ℝ : 0 ≤ x ≤ 1}. Pick two i.i.d. random variables B_{1}, B_{2}, uniformly distributed in this interval. These represent the break points. Let U = B_{1} ∧ B_{2}, V = B_{1} ∨ B_{2}. The intervals [0, U], [U, V], [V, 1] represent the three smaller sticks, and have lengths X = U, Y = V − U, Z = 1 − V. Therefore the above inequalities are written as

U ≤ 1/2
V − U ≤ 1/2
V ≥ 1/2.

We thus need to compute the probability

p = P(U ≤ 1/2, V ≥ 1/2, V − U ≤ 1/2).

Since X_{1}, X_{2} are interchangeable,

p = 2 P(X_{1} ≤ X_{2}, X_{1} ≤ 1/2, X_{2} ≥ 1/2, X_{2} − X_{1} ≤ 1/2).

Since (X_{1}, X_{2}) is uniformly distributed in the square [0, 1]^{2}, it’s obvious that p/2 is the area of the set

{(x_{1}, x_{2}) ∈ [0, 1]^{2} : x_{1} ≤ x_{2}, x_{1} ≤ 1/2, x_{2} ≥ 1/2, x_{2} − x_{1} ≤ 1/2},

which is a right isosceles triangle two sides of which have length 1/2 and thus its area is 1/8. Hence p = 1/4.

5.11 We give the intuition behind this identity in law. Imagine you have d alarms in your bedroom, each set to ring at an exponential time with rate 1 (hour^{−1}, say). The alarms function independently of one another. (This ensures that you will, at some point, get out of bed.) The first alarm will ring at a time (the minimum of d i.i.d. exponential random variables) which is exponentially distributed with rate the sum of the rates, i.e. rate d. In other words, the first alarm rings at time X_{1}/d, where X_{1} is an exponential random variable with rate 1. After the first alarm rings, there are d − 1 alarms remaining. Note two things:

(i) by the memoryless property, the remaining times are independent of the first ring and (ii) they are independent of the first ring. Therefore, the second alarm will ring at a remaining time which is exponential with rate d − 1. In other words, the second alarm rings at a time X_{2}/d − 1 (where X_{2} is an exponential random variable with rate 1) after the first ring. Thus, the second alarm rings at time distributed like

\frac{X_{1}}{d} + \frac{X_{2}}{d − 1}

1Euclid (ca. −300): Elements, Alexandria.

where X_{1}, X_{2} are i.i.d. Exp(1). Continuing in this manner, we see that the d-th alarm will ring at time

\frac{X_{1}}{d} + \frac{X_{2}}{d − 1} + ⋯ + \frac{X_{d−1}}{2} + X_{d},

where X_{1}, …, X_{d} are i.i.d. Exp(1). You can make the argument formal by first proving the analogue of Exercise 5.6, namely that if X, Y are independent positive random variables, with X being exponential, then

P(X − Y > t|X > Y) = P(X > t),

for all t > 0, and then by using induction.

5.12 This follows immediately from Exercise 3.4. But let’s prove it directly.

P(− ln U/λ > t) = P(U < e^{−λt}) = e^{−λ},

because U is uniform.

5.13 This is called regenerative property of the Gamma function. We have

Γ(β) = ∫_{0}^{∞} y^{β−1}(−e^{−y})′dy = [y^{β−1}e^{−y}]_{0}^{∞} − ∫_{0}^{∞}(−e^{−y})(y^{β−1})′dy
= ∫_{0}^{∞} e^{−y}(β − 1)y^{β−2}dy = (β − 1)Γ(β − 1).

A few remarks on rigour: We take β > 1, so that is why the value of y^{β−1}e^{−y} at y = 0 equals 0. Second, y^{β−1}e^{−y} → 0, as y → ∞, that is why [y^{β−1}e^{−y}]_{0}^{∞} = 0. Third, all integrals in the derivation above converge.

5.14 We have

Γ(1/2) = ∫_{0}^{∞} y^{−1/2}e^{−y}dy.

Change variable by

y = x^{2}/2

so that

y^{−1/2} = \frac{\sqrt{2}}{x}, e^{−y} = e^{−x^{2}/2}, dy = xdx.

Then

Γ(1/2) = \sqrt{2} ∫_{0}^{∞} e^{−x^{2}/2} = \sqrt{2}\sqrt{2π} \frac{1}{\sqrt{2π}} ∫_{0}^{∞} e^{−x^{2}/2} = 2\sqrt{π} P(N > 0),

where N is a standard normal random variable. So P(N > 0) = 1/2, and the result follows.

5.15 Here X has law Γ(β, λ), where β, λ > 0. Picking θ small enough so that the integral below converges (we will see later how small), we have

M_{X}(θ) = ∫ e^{θx} \frac{λ^{β}}{Γ(β)}x^{β−1}e^{−λx}dx

= \frac{λ^{β}}{Γ(β)} ∫ x^{β−1}e^{−(λ−θ)x}dx

= \frac{λ^{β}}{Γ(β)} \frac{1}{(λ − θ)^{β}} ∫ y^{β−1}e^{−y}dy

= \frac{λ^{β}}{Γ(β)} \frac{1}{(λ − θ)^{β}} Γ(β)

= (\frac{λ}{λ − θ} )^{β} .

Looking at the second line above, we see that the integral converges for all θ ∈ (−∞, λ), and diverges if θ ≥ λ. Compute a couple of derivatives:

M′(θ) = β (\frac{λ}{λ − θ} )^{β−1} \frac{λ}{(λ − θ)^{2}}

M′′(θ) = (\frac{λ}{λ − θ} )^{β} β^{2} (λ − θ)^{−2} + (\frac{λ}{λ − θ} )^{β} β (λ − θ)^{−2}

Set θ = 0:

E X = M′(0) = \frac{β}{λ}, E X^{2} = \frac{β^{2}}{λ^{2}} + \frac{β}{λ}.

5.16 Let (see Exercise 5.15)

M_{β,λ}(θ) = (\frac{λ}{λ − θ} )^{β}

be the generating function of a Γ(β, λ) probability measure. Then

M_{β_{1},λ}(θ)M_{β_{2},λ}(θ) = (\frac{λ}{λ − θ} )^{β_{1}+β_{2}} = M_{β_{1}+β_{2},λ}(θ),

and this proves the claim, because the generating function (being a Laplace transform) characterises the measure.

5.17 By changing variable t = x/\sqrt{2}, it is enough to prove that

I := ∫_{−∞}^{∞} e^{−t^{2}}dt = \sqrt{π},

But

I^{2} = (∫_{−∞}^{∞} e^{−x^{2}}dx) (∫_{−∞}^{∞} e^{−y^{2}}dy)

and, by Fubini’s theorem,

I^{2} = ∫_{ℝ^{2}} e^{−(x^{2}+y^{2})}d(x, y).

Since the function to be integrated is rotationally invariant on the plane, we use polar coordinates, i.e.

x = r cos θ, y = r sin θ, r > 0, 0 ≤ θ < 2π.

Since x^{2} + y^{2} = r^{2}, and since d(x, y) = rd(ρ, θ), we have

I^{2} = ∫_{ℝ^{2}} e^{−r^{2}}rd(r, θ).

Using Fubini’s theorem once more,

I^{2} = ∫_{0}^{∞} dr r e^{−r^{2}} ∫_{0}^{2π} dθ = 2π ∫_{0}^{∞} \frac{1}{2}se^{−s}ds = π.

So, clearly,^{2} I = \sqrt{π},

5.18 First, complete the square in the exponent:

\frac{1}{2}x^{2} − θx = \frac{1}{2}(x^{2} − 2θx + θ^{2} − θ^{2}) = \frac{1}{2}(x − θ)^{2} − \frac{1}{2}θ^{2}.

Therefore,

M_{X}(θ) = \frac{1}{\sqrt{2π}} ∫_{−∞}^{∞} e^{− \frac{1}{2} (x−θ)^{2}+ \frac{1}{2} θ^{2}}dx

= e^{\frac{1}{2} θ^{2}} \frac{1}{\sqrt{2π}} ∫_{−∞}^{∞} e^{− \frac{1}{2} (x−θ)^{2}}dx

= e^{\frac{1}{2} θ^{2}} \frac{1}{\sqrt{2π}} ∫_{−∞}^{∞} e^{− \frac{1}{2} y^{2}}dy

= e^{\frac{1}{2}} ,

because the last integral is the integral of a standard normal density, and hence it equals 1.

5.19 Recall the “stability property” of the Gamma distribution, namely, if X_{β_{j},λ}, j = 1, 2 are independent random variables with laws Γ(β_{i}, λ), respectively, then X_{β_{1},λ} + X_{β_{2},λ} has law Γ(β_{1} + β_{2}, λ). Here, Y_{1}^{2}, Y_{2}^{2} are independent, both with law Γ(1/2, 1/2). Hence Y_{1}^{2}+Y_{2}^{2} has law Γ(1, 1/2) = Exp(1/2).

5.20 This requires reviewing the material you learnt and just summarising in an artsy manner.

2Lord Kelvin (1824-1907) was an admirer of Joseph Liouville. It is said that, one day, while Kelvin was lecturing in Glasgow, he asked his class: ‘Do you know what a mathematician is?’ He then wrote the following equation on the blackboard

∫_{−∞}^{∞} e^{−x^{2}}dx = \sqrt{π}, and said: ‘A mathematician is one to whom that is as obvious as that 2 × 2 = 4 is to you; Liouville was a mathematician.’

5.21 Let U = X + 2Y, V = 3X − 4Y. We want to compute E(U|V). We know that 1) E(U|V) = cV, for some constant c,

2) U − E(U|V) is independent of V. Therefore U − cV and V are uncorrelated:

0 = E((U − cV)V) = E UV − c E V^{2},

whence

c = \frac{E UV}{E V^{2}} .

We have

E UV = E(X + 2Y)(3X − 4Y) = 3 E X^{2} − 8 E Y^{2} + 0 = 3 − 8 = −5,
E V^{2} = E(3X − 4Y)^{2} = 9 E X^{2} + 16 E Y^{2} + 0 = 25.

So c = −5/25 = −1/5, and so E(U|V) = −V/5.